An empirical study using permutation-based resampling in meta-regression
نویسندگان
چکیده
BACKGROUND In meta-regression, as the number of trials in the analyses decreases, the risk of false positives or false negatives increases. This is partly due to the assumption of normality that may not hold in small samples. Creation of a distribution from the observed trials using permutation methods to calculate P values may allow for less spurious findings. Permutation has not been empirically tested in meta-regression. The objective of this study was to perform an empirical investigation to explore the differences in results for meta-analyses on a small number of trials using standard large sample approaches verses permutation-based methods for meta-regression. METHODS We isolated a sample of randomized controlled clinical trials (RCTs) for interventions that have a small number of trials (herbal medicine trials). Trials were then grouped by herbal species and condition and assessed for methodological quality using the Jadad scale, and data were extracted for each outcome. Finally, we performed meta-analyses on the primary outcome of each group of trials and meta-regression for methodological quality subgroups within each meta-analysis. We used large sample methods and permutation methods in our meta-regression modeling. We then compared final models and final P values between methods. RESULTS We collected 110 trials across 5 intervention/outcome pairings and 5 to 10 trials per covariate. When applying large sample methods and permutation-based methods in our backwards stepwise regression the covariates in the final models were identical in all cases. The P values for the covariates in the final model were larger in 78% (7/9) of the cases for permutation and identical for 22% (2/9) of the cases. CONCLUSIONS We present empirical evidence that permutation-based resampling may not change final models when using backwards stepwise regression, but may increase P values in meta-regression of multiple covariates for relatively small amount of trials.
منابع مشابه
Author's response to reviews Title: An empirical study using permutation based resampling in meta-regression Authors:
متن کامل
A Gentle Introduction to Resampling Techniques
2 Overview of resampling 2 Permutation Methods 3 Bootstrapping 3 Monte Carlo 4 Failure of t-test: Girl Scout cookie sales 4 Demonstration of permutations 5 Demonstration of Howell‟s resampling programs 5 Howell‟s example: Hurry up, Fella! 5 parametric t 6 Permutation tests (also called Randomization tests) 6 t as a measure of an effect 6 Howell‟s Resampling program 7 Howell‟s Randomization (per...
متن کامل146-31: METAPERM2: A SAS® Macro for Permutation Tests of Linear Models in Fixed and Random Effects Meta-Analyses
With the growing popularity of meta-analytic techniques to analyze and synthesize results across sets of empirical studies, has come concern about the sensitivity of traditional tests in meta-analysis to violations of assumptions. Such violations are particularly distressing because the tenability of such assumptions in primary studies is often impossible to evaluate. Permutation tests for line...
متن کاملMETAPERM: A SAS Macro for Permutation Tests of Linear Models in Meta-analysis
With the growing popularity of meta-analytic techniques to analyze and synthesize results across sets of empirical studies, has come concern about the sensitivity of traditional tests in meta-analysis to violations of assumptions. Such violations are particularly distressing because the tenability of such assumptions in primary studies is often impossible to evaluate. Permutation tests for line...
متن کاملA Simulation Study to Compare the Performance of Permutation Tests for Time by Group Interaction in an Unbalanced Repeated-Measures Design, Using Two Permutation Schemes
The use of permutation tests for the analysis of data incorporating repeated measurements presents a choice of possible permutation schemes to generate the empirical null distribution of the test statistic. A commonly-recommended scheme keeps the vector of observations for each subject intact, with permutation based on group membership only. Another option is to permute the observations across ...
متن کامل